Method for image product recommendation

ABSTRACT

Embodiments of the present disclosure a method for determining product relevancy including extracting metadata from an image file of a digital image collection, the metadata being indicative of at least one feature of the image file. The method includes creating an input profile corresponding to the metadata extracted from the image files of the digital image collection. The method includes comparing the input profile to a product profile, the product profile having one or more rules corresponding to a consumer product, wherein the rules are indicative of the requirements of the product. The method includes determining a match score, the match score indicative of a relevancy of the product profile to the input profile such that a high relevancy correlates to a consumer product that is suited to the input profile and a low relevancy correlates to the consumer product that is not suited to the input profile.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims benefit of U.S. Provisional Application No.62/273,641 filed Dec. 31, 2015, entitled “Method for Image ProductRecommendation,” the entirety of which is incorporated herein.

BACKGROUND OF THE INVENTION 1. Field of the Invention

The present invention relates to image processing and, moreparticularly, to suggesting products based on the contents of an image.

2. Description of Related Art

Consumer products may include items incorporating one or more images(e.g., photographs) captured by a consumer for personal or professionaluse. For example, the consumer product may include an image on at-shirt, coffee mug, calendar, or the like. Moreover, collections ofphotographs may be organized and compiled in a photo album (e.g., adigital photo album, a digital slide show, a hard copy print out) forlater viewing. Users may present images for processing (e.g., at a photokiosk) and be presented with one or more options for additional productsthat may be purchased incorporating the images. For example, the kioskmay suggest placing an image on a t-shirt. However, often thesuggestions are not relevant to the image. For example, the kiosk maysuggest placing an unclear, out of focus image on the t-shirt, or toplace an image that may not be very important to the consumer on thet-shirt. Additionally, the kiosk may suggest placing a collection ofimages in a photo book where the total number of images is too small tofill out the photo book. As a result, it is now known that a method forpresenting relevant consumer products to users is desirable.

U.S. Pat. No. 8,934,717, issued Jan. 13, 2015, to Newell et al.describes a method for selecting the most suitable assets from an imagecollection, given an image product. The image product is either chosenby the user or picked by the system based on an “identified triggeringevent” such as Mother's Day or New Year's Day. For example, a collectionof vacation images from a beach presented to the system before Mother'sDay would be treated as content for a Mother's Day product even thoughthey may not be relevant to that product, since no semantic analysis ofthe images is done to determine suitability.

U.S. Pat. No. 8,756,114, issued Jun. 17, 2014, to Fredlund et al.describes a method for generating a product recommendation using theuser's images, but the rules used for recommending the product are basedon user profile information, contextual information, triggering eventsetc. For example, a product offering may be based on the consumer beingover a certain age, or consumer's interest in a particular hobby, or acurrent popular sporting event. However, this is an example of checkingif a product can be generated, and not a method for optimizing theproduct selection to the image collection to produce a list of possibleproducts, ranked by suitability to the given image collection, asdisclosed in our application.

Cok has described associating an image-type distribution based on imagecontent to a theme (e.g., birthday party) in U.S. Pat. No. 8,831,360issued Sep. 9, 2014; and a method for matching images from a collectionto the image-type distribution associated to the theme in U.S. Pat. No.8,917,943 issued on Dec. 23, 2014. However, no method for thenon-trivial step of automatic product selection or theme selection hasbeen disclosed in either invention. Furthermore, it should be noted thatthe theme as described in these patents is not equivalent to arecommended product type (e.g., calendar, photobook, mug) as defined inthe present invention.

In U.S. Pat. No. 8,611,677 issued Dec. 17, 2013, Das et al havedescribed a method for classifying images or videos in a digital imagecollection into one of several event categories, using a combination oftime-based and content-based features. The objective is to allow theconsumer to search for and browse images in the collection depictingspecific events, and using the event category labels to enable theautomated generation of event-specific creative media outputs. Quotingfrom their disclosure “For example, a vacation in Europe will suggestthe use of a relevant background design and theme that reflects thecultural and regional characteristic of the location where the eventtook place. A party event will evoke the use of a fun and whimsicaltheme and mood for the album.” It is clear that the aim there is to usethe event label(s) to select the appearance of an output compositionusing images in the event. This provides a useful template forconstructing, for example, a particular page on a photobook. However, itdoes not teach how to recommend a specific product category such as acalendar, a photobook, or a collage, based on the semantic content orevent information of the captured images.

BRIEF SUMMARY OF THE INVENTION

In an embodiment a method for determining product relevancy includesextracting metadata from an image file of a digital image collection,the metadata being indicative of at least one feature of the image file.The method also includes creating an input profile corresponding to themetadata extracted from the image files of the digital image collection.The method further includes comparing the input profile to a productprofile, the product profile having one or more rules corresponding to aconsumer product, wherein the rules are indicative of the requirementsof the product. The method also includes determining a match score, thematch score indicative of a relevancy of the product profile to theinput profile such that a high relevancy correlates to a consumerproduct that is suited to the input profile and a low relevancycorrelates to the consumer product that is not suited to the inputprofile.

In a further embodiment a method to determine and display relevantconsumer products related to image files includes providing an imageprocessing system to a user on a user computing device. The method alsoincludes receiving the image files at a server, the server including oneor more processors and a memory that stores product profiles related toconsumer products. In embodiments, the one or more processors operate toproduce, responsive to the received image files, an input profiledefined at least in part by extracting metadata from the image files.The one or more processors also compare the input profile to the productprofiles related to consumer products, the product profiles includingone or more elements corresponding to the metadata of the input file.The one or more processors further determine a match score by applying areward or penalty based on the correlation of the image profile to theproduct profile, the reward increasing the value of the match score.Also, the one or more processors output, to the image processing system,a list of the consumer products determined relevant to the input profilebased on the value of the match score.

In an embodiment a non-transitory computer-readable medium withcomputer-executable instructions stored thereon executed by one or moreprocessors to perform a method to determine relevancy of a consumerproduct based on an image file includes generating an input profile byextracting at least one metadata feature from the image file. The methodalso includes comparing the input profile to a product profile, theproduct profile having at least one element corresponding to the atleast one metadata feature. The method further includes calculating amatch score based on the comparison between the input profile and theproduct profile, wherein a positive comparison between the input profileand the product profile increases the value of the match score. Themethod also includes outputting a list of consumer products based on thevalue of the match score, the list comprising products having arespective match score above a threshold value.

BRIEF DESCRIPTION OF THE FIGURES

FIG. 1 is a perspective view of an embodiment of an image processingunit;

FIG. 2 is a schematic view of an embodiment of a user interface;

FIG. 3 is a schematic view of an embodiment of an image file containinga photographic image, in accordance with the present disclosure;

FIG. 4 is a schematic diagram of an embodiment of an image profile;

FIG. 5 is a flow chart of an embodiment of an image file evaluationmethod;

FIG. 6 is a table of an input profile;

FIG. 7 is an table of an input profile;

FIG. 8 is a table of an input profile;

FIG. 9 is a schematic view of an embodiment of consumer products;

FIG. 10 is a schematic diagram of an embodiment of a product profile;

FIG. 11 is a table of a product profile;

FIG. 12 is a flow chart of an embodiment of a method for calculating amatch score;

FIG. 13 is a flow chart of a method for evaluating an input profiledatabase; and

FIG. 14 is a schematic diagram of an embodiment of a computing system.

DETAILED DESCRIPTION OF THE INVENTION

Embodiments of the present disclosure include a method for evaluatingand presenting a list of consumer products related to an image file or acollection of image files based on properties of the image file orcollection of image files and the consumer products. For example, themethod may evaluate metadata of the image file (both recorded andderived) to generate an image profile of the image file includinginformation such as the time and date of creation, location, presence ofhuman faces, and the like. Accordingly, the image profile includes oneor more elements of the image file that may be evaluated against theconsumer products to determine a relevant product based on the imageprofile. In certain embodiments, an input profile is generated bycombining the image profiles from a collection of images. Consumerproducts available on the system are described by product profiles. Forexample, the product profile may include rules pertaining to themeelements, preferred number of photos, time of year, and the like tocharacterize the consumer product. Furthermore, the method includescomparing the input profile to the product profile and calculating amatch score to determine if the product profile corresponds to aconsumer product that is relevant to the given input profile. Forexample, the method may evaluate the input profile against the productprofile and add weighted values to the match score when features desiredin the product profile are present in the input profile, and to subtractweighted values to the match score when the features are not present inthe input profile. As a result, the match score can be used to determinethe relevance of the consumer product to the image files presented tothe system. In this manner, the method may present only relevantconsumer products for review to users.

Turning to FIG. 1 , a perspective view of an embodiment of an imageprocessing unit 10 having an image processing system 12 being utilizedby a user 14 (e.g., a consumer) is shown. The imaging processing unit 10includes a display 16 arranged on top of a body 18. In the illustratedembodiment, the display 16 includes an interface 20 that receives inputsfrom the user 12 via a touch-screen. For example, as will be appreciatedby one skilled in the art, the user 14 may touch one or more inputbuttons 22 to select a variety of options (e.g., input, payment, help,etc.). While the illustrated embodiment includes the touch-screeninterface 20 on the display 16, in other embodiments the user 14 mayutilize a personal electronic device 24 (e.g., a smartphone, a personaldigital assistant, a remote, etc.) to interface with the imageprocessing unit 10. For example, a personal electronic device 24 mayinclude one or more programs (e.g., applications) having instructionsthat are executed by a readable memory of the personal electronic device24 and interface with the image processing unit 10. In this manner, theuser 14 may direct the image processing unit 10 to perform one or morepre-programmed tasks.

Returning to the body 18, a data port 26 is positioned on the body 18and receives information related to images (e.g., photographs, videos,etc.). For example, the data port 26 (e.g., USB, SD-card, CD-ROM, etc.)may interface with a memory card 28 (e.g., flash memory, CD-ROM, etc.)when the memory card 28 is positioned into readable contact with thedata port 26. Moreover, in certain embodiments, the data port 26 mayinclude wireless data transfer capabilities (e.g., Wi-Fi, BLUETOOTH, 4G,cellular, etc.) to enable transfer to the data port 26 from the personalelectronic device 24 without forming a physical connection with the dataport 26. Further, in the illustrated embodiment, the body 18 includes adispenser 30 to provide products to the user 14 and a payment module 32to receive payment (e.g., cash, credit cards, wireless payment systems,etc.) from the user 14. In this manner, the user 14 may interface withthe image processing unit 10 to provide images for processing.

FIG. 2 is a schematic view of an embodiment of a user interface 40 foruploading photos for analysis by the image processing system 12. Incertain embodiments, the user interface 40 may be utilized by thedisplay 16 of the image processing unit 10. However, while FIG. 1illustrated an image processing unit 10 (e.g., a kiosk) for receivingand processing the images, in other embodiments the user 14 may interactwith the user interface 40 away from the image processing unit 10. Forexample, the user interface 40 may be an embodiment of a web site forreceiving the image files for later processing. Furthermore, in certainembodiments, the user interface 40 may correspond to an applicationloaded on to the personal electronic device 24 to enable the user 14 toupload the prepare photos for later use via the image processing unit10, a photo processing center, or electronically.

In the illustrated embodiment, the user interface 40 includes an imagewindow 42 that includes a list of image files currently added (e.g.,uploaded to) the image processing system 12. For example, the imagefiles may be uploaded via the input button 44 which may receive thefiles from the user's hard drive (e.g., RAM, ROM, optical discs, flashmemory, etc.), the personal electronic device 24, the memory card 28, orthe like. Further, the image processing system 12 may receive the imagefiles from social media sites (e.g., FACEBOOK, INSTAGRAM, TWITTER, andthe like) via linking buttons 46 that direct the user 14 to enableaccess to the respective social media site. Accordingly, image files maybe added (e.g., interfaced with, uploaded to) the image processingsystem 12 without being stored on local memory, thereby increasing theconvenience and access of the image processing system 12 to the user 14.

In the illustrated embodiment, the user interface 40 includes a productwindow 48 to display consumer products 50 that may be selected by theuser 14 for purchase and/or interactive use. For example, the productsmay include photo albums, social media banners, greeting cards, birthannouncements, or the like and include one or more of the image filesuploaded to the image window 42. As will be discussed in detail below,the image processing system 12 evaluates the image files to generate aninput profile 110 to compare with a product profile corresponding to theproducts 50. By comparing the input profile to the product profile, theproducts 50 in the product window 48 may be arranged such that the mostrelevant (e.g., products 50 most likely to be used and/or purchased bythe user 14) are arranged first. As a result, the user 14 may be morelikely to utilize and/or purchase the products 50 because the user 14will not search through several products 50 which do not correspond tothe uploaded image file. Moreover, the products 50 may be specificallytailored to the image files (e.g., similar themes), and thereby providethe user 14 with ideas and/or options to display or use the image files.

FIG. 3 is a schematic view of an embodiment of an image file 60 formedat least in part by a still photographic image 62. In the illustratedembodiment, the still photographic image 62 depicts a family 64positioned near a Christmas tree 66. As shown, the family 64 includesfour humans and one animal. Further, the depicted photographic image 62includes presents 68 under the Christmas tree 66. However, in theillustrated embodiment, the image file 60 includes more than thephotographic image 62. For example, the image file 60 includes metadatacorrelating a variety of information such as time, date, GPS location,image quality, camera-type used to create the image file 60, and thelike. In the illustrated embodiment, time and date metadata 70 ispositioned at the bottom right corner of the photographic image 62.Further, in certain embodiments, the image file 60 includes locationmetadata 72. The location metadata 72 may include GPS coordinatesrelated to the location where the photographic image 62 was taken. Inthe illustrated embodiment, the location metadata 72 is depicted in thetop left corner, however, the location metadata 72, as well as the timeand date metadata 70, may not be physically visible on the photographicimage 62. That is, the time and data metadata 70 and the locationmetadata 72 may be electronically embedded in the image file 60. Assuch, the time and date metadata 70 and the location metadata 72 may bereferred to as recorded metadata (e.g., capture-based metadata) becausethe information is recorded by the image file 60 at the time the imagefile 60 is generated.

Moreover, derived metadata may be obtained from analysis of thephotographic image 62. As used herein, derived metadata refers toinformation obtained from analyzing one or more areas of thephotographic image 62 for one or more distinguishing features. In theillustrated embodiment, derived metadata includes at least content-basedmetadata, face-based metadata, and event-based metadata. As will bedescribed below, derived metadata may be combined with recorded metadatato generate an image profile related to the image file 60.

Content-based metadata 74 refers to features that are computed fromimage pixels and that are intended to be an indicator of image content.In other words, evaluation of content-based metadata utilizes analysisof the image pixels forming the photographic image 62 of the image file62 to extract features indicative of one or more characterizing contentprofiles. For example, content-based metadata 74 may be related to thebackground and/or scenery of the photographic image 62. As such,photographic images of leaves, mountains, a beach scene, or the like maybe extracted and analyzed utilizing content-based metadata. Suchtechniques have been described in commonly assigned U.S. Pat. No.6,504,951 and U.S. Patent Publication No. 2005/0105776, both of whichare hereby incorporated by reference in their entireties.

In the illustrated embodiment, content-based metadata 74 may be derivedfrom the image pixels of the photographic image 62. For example, theChristmas tree 66 may be extracted from the image pixels and analyzed.Further, other features of the background 70 may be evaluated todetermine the context of the photographic image 62 (e.g., indoors,outdoors, holiday decorations, morning, evening, etc.). In theillustrated embodiment, analysis of the Christmas tree 66, among otherfeatures, may enable the content-based metadata 74 to determine thephotographic image 62 is an indoor photograph and a holiday scene.

Face-based metadata refers to analysis and detection of human faces fromimage pixels. For example, in the illustrated embodiment, face-basedmetadata 76 extracts the faces of the four human beings in thephotographic image 62. As such, the later generated image profile mayinclude information indicative that the image file 60 contains at leastone human face. Face detection algorithms and methods for detectinghuman faces are well known in the art of digital processing andtherefore are not discussed in greater detail.

Event-based metadata refers to features of the image file 60 detected onan event level. For example, the event level may include the temporalduration of the event, the number of images in the event (e.g., viaanalysis of a photo album created on social media, via analysis ofseveral image files 60 created over a time period), event user tags(e.g., via analysis of hash tags or captions on social media), eventcategory information, or the like. Moreover, the event-based metadatamay utilize the time and data metadata 70 to analyze the time of year ofthe event to correlate the event to known holidays. For example, imagefiles 60 generated on December 25 may be indicative of photographsassociated with Christmas. Accordingly, event-based metadata may beutilized to form the image profile.

FIG. 4 is a schematic diagram of an embodiment of an image profile 90generated by the image processing system 12 after analyzing the imagefile 60. In certain embodiments, the image profile 90 consists of anaggregate of the recorded metadata and the derived metadata for eachimage file 60. That is, the image profile 90 includes the featuresextracted from the image pixels while the image file 60 is analyzed bythe image processing system 12. In the illustrated embodiment, the imageprofile 90 a may correlate to analysis performed of an individual imagefile 60. For example, for the photographic image 62 depicted in FIG. 3 ,the image profile 90 a may include features extracted from the imagefile 60 corresponding to the time and data metadata 70, the locationmetadata 72, the content-based metadata 74, the face-based metadata 76,and event-based meta-data 92. Accordingly, the image profile 90 a for agiven image file 60 corresponding to a photograph or video includes atleast one feature extracted from the image file 60 corresponding atleast in part to the derived metadata or the recorded metadata.

In certain embodiments, each individual image file 60 may have the imageprofile 90 associated with the given image file 60. However, inembodiments where large numbers of image files 60 are analyzed (e.g.,100 image files, 500 image files, 1000 image files, etc.), the imageprocessing system 12 may arrange the image files 60 into one or morerelational databases of image profiles 90 correlated to an inputprofile. As will be described below, by querying a database containingthe image profiles, processing resources may be conserved becausecertain image profiles within the input profile may not be evaluated ifthey do not correspond to (e.g., match) the query. Furthermore, incertain embodiments, user inputs may be utilized to form at least aportion of the image profile 90. For example, the image processingsystem 12 may prompt the user to input information regarding desirableproducts, such as age of the recipient, culture of the recipient, andthe like. These inputs may be utilized to determine suitable consumerproducts via the image processing system 12.

FIG. 5 is a flow chart of an embodiment of a method 100 for extractingmetadata from a digital image collection 102 (e.g., one or more imagefiles 60). The digital image collection is processed by an eventdetector (block 104). For example, the event detector clusters imagesbased on the same event via algorithms, as disclosed in commonlyassigned U.S. Pat. Nos. 6,606,411 and 6,351,556, each of which arehereby incorporated by reference in its entirety. As such, the eventdetector may be utilized to generate the databases described above. Itshould be noted that in certain embodiments, the method 100 mayeliminate block 104. Subsequently, the digital image collection 102undergoes metadata analysis (block 106). For example, in the illustratedembodiment, metadata analysis includes evaluating the digital imagecollection 102 for, at least in part, the time and date metadata 70,location metadata 72, content-based metadata 74, face-based metadata 76,and event-based metadata 92. However, it is appreciated that additionalanalysis may be performed and utilized to generate the image profiles90. For example, the image quality may be evaluated to determine whetherenlarging the photographic image 62 would be appropriate withoutdistortion and/or sacrifices to clarity. The metadata extracted from theimage files 60 is output and saved (block 108) as image profiles 90.That is, target metadata may be extracted from the image files 60 togenerate the image profiles 90. In this manner, the metadata may besaved to an image profile 90 correlating to features present in thephotographic image 62 of the image file 60. In certain embodiments, theimage profiles 90 from a collection of images are combined and saved asan input profile 110. Additionally, in certain embodiments, the imageprofiles 90 of individual image files 60 may be compiled into a database(e.g., a relational database) for further analysis. As will be describedbelow, with large numbers of image files 60, arranging the imageprofiles 90 into databases may conserve computing resources because thedatabase may be queried and only image profiles 90 within the inputprofile corresponding to the query may be analyzed by the imageprocessing system 12. Accordingly, the method 100 enables the imageprocessing system 12 to analyze incoming data image collections 102(e.g., collections of image files 60, collections of video files,combinations of image files 60 and video files) for processing andevaluation for consumer product suggestion.

FIG. 6 is a table of an embodiment of an input profile 110 containing aset of image profiles 90. As used herein, input profile 110 refers to aset of image profiles 90 associated with images presented by the user 14to the system 12. In the illustrated embodiment, a first column 112includes an image identifier, a second column 114 includes featurescorrelated to content-based metadata 74, a third column 116 includesfeatures correlated to time-based metadata 70, a fourth column 118includes features correlated to face-based metadata 76, and a fifthcolumn 119 includes features correlated to event-based metadata 92. Byway of example only, in the illustrated embodiment, the image files 60may be related to a day trip at a lake to see fall foliage. As a result,the content-based metadata 74 may extract colors (e.g., red, blueyellow, fall colors), natural features (e.g., mountains, vegetation,sky, beach), and/or other features, such as indications of the outdoorsand nature generally. Accordingly, for example, the event detector 104may determine the collection of images as relates to a similar event ata similar location. Moreover, as illustrated in the input profile 110,other metadata features may be extracted from the image files, such asthe time-based metadata 70 (e.g., afternoon), the face-based metadata 76(e.g., adult female, child female, group of 2), and/or the event-basedmetadata 92 (e.g., vacation, autumn, Lake George, N.Y.). It should beappreciated that while the illustrated embodiment includes 14 images inthe input profile 110, in other embodiments, more of fewer images may bearranged into the input profile 110. For example, the input profile 110may include 10, 20, 100, 200, 300, 400, 500, one thousand, ten thousand,or any other number of images. As described above, arranging the imageprofiles 90 in the input profile 110 enables the image processing system12 to query the database of images in the input profile 110 to extractspecific image files corresponding to particular requests. For example,a query evaluating the input profile 110 for the color yellow may returnImage_1 and Image_3, in the illustrated embodiment. As a result,computing resources may be saved because each image file of the inputprofile 110 not corresponding the query may not undergo additionalanalysis.

FIG. 7 is a table of an embodiment of the input profile 110. By way ofexample only, the input profile 110 in FIG. 7 corresponds to acollection of image files 60 created in mid-December, before theChristmas holidays. For example, the photographic image 62 of FIG. 3 maybe included in the digital image collection 102 utilized to generate theinput profile 110. As shown, each image file 60 includes metadatafeatures, such as content-based metadata 74 (e.g., red, indoors), timeand data metadata 70 (e.g., morning, weekend, winter), face-basedmetadata 76 (e.g., 3 or more, adult male, child female), and event-basedmetadata 92 (e.g., short duration, social moment, home). As such, theindividual image profiles 90 for the respective image files 60 may bearranged into the input profile 110 for evaluation by the imageprocessing system 12.

FIG. 8 is a table of an embodiment of the input profile 110. In theillustrated embodiment, the input profile 110 includes over two hundredimage files 60. However, as described above, in other embodiments theinput profile 110 may include more or fewer image files 60. As shown,the image files 60 incorporated with the input profile 110 are selectedfrom the digital image collection 102 including at least 260 images,because the first image file 60 is listed as Image_1 and the last imagefile 60 is listed as Image_260. By way of example only, in theillustrated embodiment, the image files 60 are from a family vacation atan amusement park in Orlando, Fla. As shown, by evaluating the imagefiles 60, the input profile 110 includes metadata corresponding tocontent-based metadata 74, time and date metadata 70, face-basedmetadata 76, and event-based metadata 92. For example, Image_1 includesfeatures such as sky, buildings, the outdoors, morning, weekday, 3 ormore faces, small faces, child female faces, and a location in Orlando,Fla. In this manner, the input profiles 110 may contain the metadata formultiple image files 60 for evaluation by the image processing system10.

FIG. 9 is a schematic view of an embodiment of a collection of consumerproducts 120 that may be sold by and/or electronically generated by theimage processing system 12. For example, the consumer products 120 maybe ordered from the image processing unit 10 and/or from a web siteincorporating the image processing system 12. In certain embodiments,the consumer products 120 may be physical goods, such as cards, photoalbums, coffee cups, t-shirts, or the like. Further, in certainembodiments, the consumer products 120 may be electronic goods such ase-cards, slide shows, digital advertisements, or the like. Therefore, asused herein, consumer products 120 refers to any physical or electronicitem that may incorporate one or more image files 60 and/or video files.

In the illustrated embodiment, the consumer products 120 include a card122 (e.g., a Christmas card, a wedding announcement, a birthannouncement, a birthday card, etc.), a coffee cup 124, a photo book126, and a slide show 128. However, other products 120 includingcalendars, posters, and the like may also be provided. As describedabove, the image processing system 12 incorporates product profilescorresponding to the consumer products 120. For example, the imageprocessing system 12 may include pre-loaded instructions for each of theconsumer products 120 available via the image processing system 12.Further, in certain embodiments, the image processing system 12 mayanalyze the consumer products 120 and generate a series of rules and/orfeatures indicative of the consumer products 120. That is, the imageprocessing system 12 may extract data from the products 120 whichcorrelate to features of image files. As such, the consumer products 120may be compared to the image profiles 90 to determine suitable consumerproducts 120 based on the image profiles 90.

For example, in the illustrated embodiment, the card 122 may include aphotographic space 130 to include the photographic image 62 of the imagefile 60. Furthermore, the card 122 may include a text space 132corresponding to a message being relayed by the card 122. In theillustrated embodiment, the card 122 is a Christmas card and, as aresult, the photographic space 130 may correspond to the photographicimage 62 including certain properties related to Christmas, such as aChristmas tree, a group of human beings, the color red, gifts, and thelike. As will be described below, the features corresponding to the card122 may be translated as a set of rules and/or weighing factors to becompared to the image profiles 90.

Still further, the coffee cup 124 includes the photographic space 130.In certain embodiments, the photographic space 130 on the coffee cup 124may correspond to certain properties of the image file 60, such as ahuman face, a nature scene, or the like. As such, the product profilefor the coffee cup 124 may be established to evaluate the image profile90 for the above-mentioned features.

Moreover, the illustrated photo book 126 includes photographic spaces130 and text spaces 132. For example, the photographic spaces 130 mayinclude photographic images 62 related to one another via the locationmetadata 72. That is, the photographic images 62 may all be taken at thesame location (e.g., during the same event), thereby providing cohesionand unity to the photo book 126. Furthermore, the illustrated photo book126 includes theme elements 134. The theme elements 134 are related tothe photographic images 62, thereby further bringing unity to the photobook 126. For example, if each photographic image 62 includesphotographic images 62 having location metadata 72 indicative of a themepark (e.g., DISNEY WORLD, SIX FLAGS, etc.), then the theme elements 134may corresponding to the theme park (e.g., characters, colors, etc.). Asa result, the photo book 126 is more relevant to the needs of the user14 because the theme shown in the photographic images 62 is alsoreflected on the pages via the theme elements 134. Additionally, theproduct profile related to the photo book 126 may include the themeelements 134 as a factor when evaluating respective image profiles 90.

In the illustrated embodiment, the slide show 128 includes threephotographic images 62. As will be appreciated, the photographic images62 may be related to one another to form a unified, coherent themethroughout the slide show. For example, the product profile related tothe slide show 128 may evaluate the time and data metadata 70 of theimage files 60 to place the photographic images 62 in sequential order,thereby illustrating the progression of an event over time. Further,because the slide show 128 includes more than one image file 60, theimage processing system 12 may exclude the slide show 128 as an optionwhen only one image file 60 is being evaluated. As a result, morerelevant consumer products 120 may be displayed to the user 14.

FIG. 10 is a schematic diagram of an embodiment of a product profile 140corresponding to one or more consumer products 120. As used herein, theproduct profile 140 refers to a series of rules and/or elements (e.g.,features) corresponding to one or more consumer products 120. Forexample, the product profile 140 for a given consumer product 120 maycorrelate to elements (e.g., colors, images, scenery, time of year,holiday, etc.) related to the consumer product 120. As will be describedbelow, evaluating the image files 60 against the product profiles 140enables the image processing system 12 to present the most relevantconsumer products 120 to the user 14, thereby improving sales of theconsumer products 120 and reducing the frustration and difficulty to theusers 14 of looking through consumer products 120 that may not berelevant to given image files 60.

In the illustrated embodiment, the product profile 140 includes rulescorrelating to the metadata (e.g., derived and recorded) that may beextracted from the image files 60. For example, in the illustratedembodiment, a first rule 142 correlates to time and date metadata 70. Incertain embodiments, the first rule 142 may be a threshold. For example,if the product profile 140 is directed toward a Christmas card, thefirst rule 142 may compare (e.g., via instructions executable on amemory via a processor) the time of year the image file 60 was createdbecause image files 60 created in the summer months (e.g., June, July,and August in the Northern Hemisphere) generally do not correlate toChristmas cards. Further, in other embodiments, the first rule 142 mayevaluate the time of the year when the consumer product 120 is beingrequested. Returning to the example of the Christmas card consumerproduct 122, the first rule 142 may determine that the Christmas cardconsumer product 122 may not be relevant to the user 14 in Februarybecause it would be approximately ten months before Christmas.Accordingly, the first rule 142 may evaluate time and date metadata 70and/or information regarding the date of purchase to determine whetherthe consumer product 120 corresponding to the product profile 140 isrelevant to the user 14.

In the illustrated embodiment, a second rule 144 correlates to locationmetadata 72. For example, the location metadata 72 may look at GPSlocations of the image file 60 to determine where the photograph wastaken. In certain embodiments, location metadata 72 proximate popularvacation sites (e.g., theme parks, skiing destinations, beaches, touristattractions, etc.) may correlate to product profiles 140 for itemsrelated to vacations, such as photo books 146. Additionally, severalimage files 60 of the digital image collection 102 located in the samearea may be indicative that the image files 60 correspond to a similarevent (e.g., a party, a sporting event, etc.) and, as a result, consumerproducts 120 which include multiple photographic images (e.g.,calendars, photo books, slide shows) may correlated to the image files60 better than consumer products 120 which include single images (e.g.,t-shirts, coffee cups 124, etc.) In this manner, the location metadatamay be utilized to provide relevant consumer products 120 to users 14.

Furthermore, a third rule 146 correlates to content-based metadata 70.For example, the content-based metadata 70 may look at elements in thephotographic image 62 to provide context to the photographic image 62.In certain embodiments, elements extracted by the content-based metadata70 such as holiday decorations, beach scenery, or the like may correlateto product profiles 140 for items related to the elements. By matchingcorresponding content-based metadata 70 with product profiles 140,specific consumer products 120 having similar themes may be provided tothe user 14. For example, content-based metadata 70 having informationidentifying mountains and ski-lifts may be utilized to suggest consumerproducts 140 having a skiing theme. In this manner, the relevance of theconsumer products 120 presented to the user 14 may be improved.

In the illustrated embodiment, a fourth rule 148 correlates toface-based metadata 76. For example, the face-based metadata 76 mayidentify the number of human faces present in the photographic image 62.In certain embodiments, face-based metadata 76 may be indicative ofevents associated with the product profiles 140. For example, face-basedmetadata 76 identifying several human faces may be indicative of a groupgathering or a party. As a result, consumer products 120 associated withlarge gatherings, such as posters or collages, may be suggested by theimage processing system 12 due to the face-based metadata 76.Furthermore, consumer products 120 that do not typically include humanfaces (e.g., landscape photographic posters) may not be presented to theuser 14.

Moreover, a fifth rule 150 correlates to event-based metadata 92, in theillustrated embodiment. In certain embodiments, user generated tags(e.g., hash tags, captions, etc.) may provide an indication as to theevent or occasion for the photo. For example, event-based metadata 92identifying hash tags related to a sporting event may be indicative of atheme or product associated with the sporting event (e.g., the specificsports team, the specific sport, etc.). Thereafter, product profiles 140corresponding to the theme associated with the event-based metadata 92may be presented to the user 14. As such, the user 14 has faster accessto relevant consumer products 120.

Furthermore, it should be appreciated that other rules and types ofanalysis may be performed in generating the image profiles 90 and/or theproduct profiles 140. For example, as described above, user inputs maybe utilized to form at least a portion of the image profile 90. Further,the product profile 140 may include one or more conditions or eventscorresponding to the user inputs. For example, the product profile 140may include cultural considerations, as described in application Ser.No. 15/167,327, filed May 27, 2016, entitled “Cross cultural greetingsystem,” which is hereby incorporated by reference. The product profile140 may evaluate the user inputs to suggest products that would bepleasing and/or not offensive to different cultures. Additionally, otherelements and factors may be evaluated. For example, if the digital imagecollection 102 included three photographs, the product profile 140associated with a calendar would not be recommended because calendarstypically include at least twelve photographs. Moreover, in certainembodiments, the image processing system 12 may analyze social media,appointment calendars, or the like to determine upcoming events (e.g.,holidays, birthdays, etc.) to provide improved selections to the user.In this manner, several properties of the image files 60 and/or theconsumer products 120 may be utilized to identify relevant consumerproducts 120 for the user 14.

As described above, in the illustrated embodiment, the product profile140 includes the first, second, third, fourth, and fifth rules 142, 144,146, 148, 150 that are compared to the image file 60 to determinerelevant consumer products 120. However, more or fewer rules may beutilized. In certain embodiments, the rules correlate to a weighingsystem that adds or subtracts points based on the result of theevaluation. For example, the rules may be Boolean statements evaluatingto TRUE/FALSE (e.g., determine whether an element is included or not) todetermine whether to add points (e.g., a reward) or to subtract points(e.g., a penalty) when determining which consumer products 120 are mostrelevant to the image files 60. For example, the first rule 142 mayevaluate whether the image file 60 was created in October, November, orDecember when evaluating if the image file 60 is appropriate for aChristmas card. Upon evaluation, an answer of TRUE (e.g., yes) may add areward (e.g., 1 point) to the weighing system to determine whether theproduct profile 140 is relevant for the image file 60. As will beappreciated, there may be more than five rules associated with eachproduct profile 140. For example, certain product profiles 140 mayinclude 1, 2, 3, 4, 6, 7, 8, 9, 10, 20, 30, 40, 50, or any suitablenumber of rules. Further, the rewards and penalties associated with therules may be varied according to importance of the rule to theassociated product 120. For example, the product profile 140 associatedwith the Christmas card 122 may place more weight on the presence of aChristmas tree or snow in the image file 60. As such, the productprofile 140 may be utilized by the image processing system 12 toevaluate consumer products 120 which may be relevant to given imagefiles 60. Moreover, in certain embodiments, the rules may be linked.That is, a TRUE result for two rules may be associated with a higherreward because two factors may be more closely linked to the productprofile 140 than individual factors.

FIG. 11 is a table of an embodiment of a catalog of product profiles140. By way of example only, the catalog includes product profiles 140corresponding to seven different consumer products 120. For example, inthe illustrated embodiment, the catalog includes a product profile 140for a Christmas card 151, a 5-image collage with a fall motif 152, a5-image collage with a winter motif 153, a coffee mug with a baby motif154, a coffee mug with a family motif 155, a photo book with a Floridatheme parks vacation motif 156, and a photo book with a Southwest USvacation motif 157. As described above, each product profile 140 hasrules corresponding to the content-based metadata 74, the time and datametadata 70, the face-based metadata 76, and the event-based metadata92. For example, the Christmas card 151 includes rules based oncontent-based metadata 74 related to the color red, the color green, andthe indoors. As shown, the red is weighted with (+1, −1), indicatingthat a TRUE response to the query whether red is present in the imagefile adds one to a match score, while a FALSE response subtracts onefrom the match score. In this manner, each rule based on metadatafeatures for each product profile 140 of a consumer product 120 may beevaluated against the input profile 110 (e.g., a database containing oneor more image profiles 90) to determine a match score indicating themost relevant product for the input profile 110. For example, the inputprofile 110 shown in FIG. 6 containing fourteen image profiles 90 of afall foliage outing may correspond strongly to the 5-image collage witha fall motif 152. As a result, upon evaluation, the 5-image collage witha fall motif 152 may be determined to be the most relevant consumerproduct 120 for the input profile 110 based on having the highest matchscore among the products in the catalog.

As can be seen in the example presented in FIG. 11 , different metadatafeatures may be more strongly weighed than other features. For example,with respect to the Christmas card 151, the face-based metadata 76feature corresponding to Groups of Three or More is weighted (+3, −2),indicating that three or more faces adds three to the match score, whileless than three faces subtracts 2. In this manner, different featuresmay be weighed more heavily to better correlate the features of theimage files 60 to the consumer products 120, thereby improving therelevance the consumer product 120 has to the various image files 60.

FIG. 12 is a flow chart of an embodiment of an evaluation method 160 fordetermining a match score of a consumer product 120 with a collection ofimages based on the metadata extracted from the image files 60. Theinput profile 110 computed using 100 is imported into the imageprocessing system 12 (block 162). For example, in certain embodiments,the input profile 110 corresponding to a single image file 60 may beevaluated. However, in other embodiments, the input profile may containone or more image files 60 incorporated into the digital imagecollection 102.

A weight representative to the evaluation is normalized (e.g., set tozero) (block 164). In the illustrated embodiment, the weight is utilizedto add and subtract the rewards and penalties associated with the rules(e.g., rules 142, 144, 146, 148, 150) incorporated with the productprofile 140. The product profile 140 is evaluated to determine if thereare pre-determined rules to evaluate (operator 166). For example, theproduct profile 140 may include a series of elements (forming rules) toevaluate against the image profile 90. In certain embodiments, the ruleis a Boolean statement evaluating to TRUE/FALSE that provides a reward(e.g., positive value) when the condition is met (TRUE), and a penalty(e.g., negative value) when the condition is not met (FALSE), and incertain embodiments, no value. For example, in certain embodiments novalue is assigned for FALSE designations because some elements may bestrong indicators of correlation with the consumer products 120, yet,the absence of the elements may not be indicative that the consumerproduct 120 does not correlate to the input profile 110. For example, inthe embodiment where the Christmas card 151 is illustrated, the presenceof snow in the input profile 110 may be a strong indicator of a winterscene, but the absence is not necessarily indicative of a non-Christmasphotographic image 62 since many photographs may be taken indoors.

After the weight is normalized, the input profile 110 is evaluatedagainst the product profile 140 (block 168). For example, in theillustrated embodiment, the analysis includes evaluating the rule (e.g.,rule 142, 144, 146, 148, 150) against each image profile 90. If the ruleis met by the image (e.g., true), then the reward is added (block 170).If the rule is not met by the input (e.g., false), then the penalty isadded (block 172). As described above, in certain embodiments thepenalty may be set to zero. The weight is computed based on the outputfrom the operator (block 174). That is, the value from the reward orpenalty is added to the weight (e.g., normalized at block 164).Thereafter, the loop 176 returns to block 166 to evaluate whether theproduct profile 140 includes additional rules (e.g., rule 142, 144, 146,148, 150). When the last rule of the product profile 140 is evaluated,the match score is determined (block 178). By way of example only, thefollowing pseudo code may be utilized to perform the evaluation method160:

match score initialized to 0.0

for each rule in product profile:

-   -   for each image profile in the input profile:        -   if the rule is met by the image profile            -   match score=match score+reward        -   else match_score=match_score+penalty    -   output match_score

The match score is a representative value that correlates to howrelevant the product profile 140 is to the input profile 110. In certainembodiments, the higher the match score, the more relevant the productprofile. Furthermore, in certain embodiments, a threshold match scoremay be utilized to only present consumer products 120 which may berelevant to users 14. As a result, the number of consumer products 120that the user 14 evaluates may be smaller, thereby simplifying thereview and likelihood the user 14 will identify relevant consumerproducts 120.

In an embodiment, image files 60 from FIG. 2 may be evaluated utilizingthe method 100 and the evaluation method 160. For example, as describedabove, features correlating to metadata of the image files 42 may beextracted. For example, the time and data metadata 70 may determine thatthe image file 60 was created (e.g., the picture was taken) in the firstweek of December. Furthermore, the location based metadata 72 maycorrelate to the family's home state, as opposed to on vacation atanother location. Furthermore, content based metadata 74 may extract theChristmas tree 66 and the presents 68. Additionally, the face-basedmetadata 76 may exact four human faces from the photograph, indicativeof a family picture. Accordingly, the image profile 90 may be createdfor later evaluation against the product profiles 140 of variousconsumer products 120. In certain embodiments, the image files 42 may beevaluated via the evaluation method 160 to select one or more consumerproducts 120. As described above, the input profile 110 may be uploadedto and the weight normalized, as illustrated in FIG. 12 . Thereafter,the product profile 140 correlating to, for example, the Christmas card122 may be evaluated against the input profile 110 to determine whetheror not a match is present. For example, the product profile 140 for theChristmas card 122 may include rules corresponding to the metadata toapply a reward or penalty based on the evaluation of the image profile.In certain embodiments, the product profile 140 may have rulescorresponding to colors (e.g., adding a reward for the presence of redor green), plants (e.g., the Christmas tree 66), and face-based metadata(e.g., family pictures). In the illustrated embodiment, the rulecorresponding to the presence of the Christmas tree 66 would beevaluated as TRUE, thereby applying a reward (e.g., +5 points) when theweight is computed. Further evaluation may evaluate the rulecorresponding to face-based metadata as TRUE, thereby also applying areward (e.g., +1 point) when the weight is computed. In this manner,each rule of the product profile 140 may be evaluated and compared tothe extracted metadata of the image file 60 to generate the match score.For example, a series of photographs with a Christmas theme maycorrelate to the photo book 126 having a winter and/or Christmas theme.In certain embodiments, high match scores correlate to consumer products120 with high relevance to the image files 42, thereby presenting theconsumer with more relevant products for purchase with less searching.For example, consumer products 120 with a match score above a thresholdmay be output to the display 16 (e.g., via a server) and displayed tothe consumer for purchase.

FIG. 13 is a flow chart of an embodiment of a database evaluation method210 utilized to evaluate input profiles 110 containing large quantitiesof image profiles 90. For example, in certain embodiments, there may beover 1,000 image profiles 90. However, the database evaluation method210 may also be utilized to evaluate input profiles 110 containing smallquantities of image profiles 90. In the illustrated embodiment, theinput profiles 90 are uploaded (block 212). For example, the image files60 added to the image processing system 12 by the user 14 may beanalyzed by the method 100 and the results saved in the input profile110. The input profile 110 may be evaluated to determine whether thereare sufficient image files 60 to be placed in a database (block 214).For example, in certain embodiments, there may be a threshold number ofimage files 60 to generate the database (e.g., 1,000 files). If thereare enough image files 60 for the database, the database may receive aquery (block 216). For example, the query may include a set ofinstructions or an interrogation, such as ‘return a list of all imageprofiles 90 satisfying rule A.’ The system may loop over the databaseand return matching image profiles 90 (block 218). Then, the systemcompiles the returned image profiles 90 to form a new input profile(block 220). Subsequently, the new input profile 110 may be evaluated(block 222) utilizing the evaluation method 160. Moreover, inembodiments where the input profile 110 is not large enough to have adatabase, the system may instruct the smaller input profile 110 to beevaluated (block 222) utilizing the evaluation method 160. In thismanner, large databases of image profiles 90 may be queried andevaluated against the product profiles 140 before the match score iscalculated, thereby limiting the evaluation to image profiles 90 likelyto correlate to the product profiles 140.

FIG. 14 is a block diagram of a computing system 190 which may beutilized by the image processing system 12. In the illustratedembodiment, the computing system 190 includes a user computing device192, such as a desktop computer, a laptop, a tablet computer, asmartphone, a personal digital assistant, the image processing unit 10,the personal electronic device 24, and the like. The user computingdevice 192 can include one or more processors 194, such as amicroprocessor, that may be used to execute machine-readable executablecode for implementing the techniques described herein. Furthermore, inthe illustrated embodiment, the user computing device 192 includes oneor more memories 196 (e.g., non-transitory computer-readable media)which may include non-volatile memory, such as read-only memory (ROM),EEPROM, and/or flash memory which may be used in conjunction withvolatile memory, such as Dynamic Random Access Memory (DRAM) and/orStatic Random Access Memory (SRAM). Further, the user computing device192 may include one or more displays 198, such as the display 16, incommunication with the one or more processors 194.

In certain embodiments, the product profiles 140 are loaded onto aserver 200 having one or more processors 202 and one or more memories204, as described above with respect to the user computing device 192.Furthermore, the server 200 can be in communication with the usercomputing device 192 (e.g., via a wired or wireless internet connection,Wi-Fi, BLUETOOTH, cellular, etc.) to send and receive informationindicative of the consumer products 120 related to the product profiles140. In certain embodiments, the one or more processors 202 of theserver 200 perform the evaluation method 160 utilizing readable codestored on the one or more memories. For example, the user computingdevice 192 may upload the image profiles 90 to the server 200 forfurther evaluation. Upon completion of the evaluation method 160, theserver 200 may relate a list of consumer products 120 to the usercomputing device 192. However, in certain embodiments the evaluationmethod 160 may be performed by the user computing device 192.

As described in detail above, embodiments of the present disclosureinclude the method 160 for evaluating and presenting a list of consumerproducts 120 related to the input profile 110 based on properties of theimage files 60 and the consumer products 120. For example, the method160 may compare metadata of the image files 60 stored in the inputprofiles 110 to product profiles 140 of the consumer products 120. Themethod 160 may assign a positive value to corresponding features and anegative value to unrelated features. As a result, the consumer products120 may be evaluated to determine whether they are relevant for thegiven set of input profiles 110. That is, the method 160 may compute thematch score to determine how well the consumer product 120 correlates tothe image files 60. As a result, the image processing system 12 may onlydisplay the relevant matches to the user 14, thereby simplifying thereview and purchase process for the user 14.

The foregoing disclosure and description of the invention isillustrative and explanatory of the embodiments of the invention.Various changes in the details of the illustrated embodiments can bemade within the scope of the appended claims without departing from thetrue spirit of the invention. The embodiments of the present inventionshould only be limited by the following claims and their legalequivalents.

The invention claimed is:
 1. A method for creating a consumer photoproduct recommendation, comprising: receiving, at an image processingunit, a digital image collection comprising a plurality of image files;extracting recorded metadata from the image files, wherein the recordedmetadata comprises location information and capture time information;extracting content-based metadata indicative of one or morecharacterizing content profiles from the image files independent ofother types of metadata, wherein the content-based metadata is derivedfrom an analysis of image pixels of a background or scenery of each ofthe image files; extracting face-based metadata from the image files,wherein the face-based metadata is derived from an analysis of imagepixels in each of the image files; extracting event-based metadataincluding a temporal duration of an event, number of images in theevent, and event user tags from the image files, wherein the event-basedmetadata is derived from an analysis of image pixels in each of theimage files; creating an input profile for the digital image collectiondefined by the extracted recorded metadata, extracted content-basedmetadata, extracted face-based metadata, and extracted event-basedmetadata; comparing the input profile to a plurality of productprofiles, wherein each product profile corresponds to a differentconsumer photo product and comprises one or more product profile rulescorresponding to each respective consumer photo product, wherein theproduct profile rules define features of each respective consumer photoproduct, and wherein each product profile rule is configured to providea reward if it is satisfied, or a penalty or no value if it is notsatisfied; determining a match score for each product profile comparedto the input profile, wherein the match score is indicative of arelevancy of the product profile to the input profile and a higher matchscore correlates to a higher relevancy, and wherein the match score isdetermined in part by evaluating whether the input profile satisfies theproduct profile rules; and displaying, on an interface of the imageprocessing unit, a rendering of the consumer photo productscorresponding to the product profiles having a match score above apre-established threshold.
 2. The method of claim 1, wherein determiningthe match score for each product profile compared to the input profilecomprises increasing the match score when a product profile rule is metby the input profile.
 3. The method of claim 1, wherein determining thematch score for each product profile compared to the input profilecomprises decreasing the match score when a product profile rule is notmet by the input profile.
 4. The method of claim 1, further comprisingreceiving the product profiles from a pre-loaded catalog on a server. 5.The method of claim 1, further comprising displaying, on the interfaceof the image processing unit, the consumer product associated with theproduct profile having the highest match score.
 6. The method of claim1, wherein displaying a rendering of the consumer photo productscomprises listing the consumer products from highest match score tolowest match score.
 7. The method of claim 1, wherein determining thematch score for each product profile compared to the input profilecomprises neither increasing nor decreasing the match score when aproduct profile rule is not met by the input profile.